1. Field of Invention
This invention relates to electronic multimedia document systems. In particular, this invention is a method and a system for providing an historical index to a multimedia document using temporal icons. More specifically, this invention provides temporal icons that present temporal information about corresponding events or timestreams and also provides complete playback control through a gestural interface on the temporal icons.
2. Description of the Related Art
The collection of audio, video, and other records of a meeting provides a rich, reviewable record of the history of a meeting or a presentation. Unfortunately, the benefits of such a record is often outweighed by the unwieldiness of sequential access. The advent of digital recordings has provided the ability to instantly access any portion of a recording. However, instant access is only valuable to the user if the user knows which portion of the recording to access and how to gain access those portions.
A user of a multimedia document system will typically want to playback the recording of a meeting to recall conversations and/or other activities that occurred during the meeting. Such a user may include one of those who were present during the meeting or may be one who did not attend the meeting.
One conventional approach to indexing a multimedia document is termed real-time note taking. Real-time note taking describes a technique whereby a person takes notes during a meeting using a system that time stamps the various individual elements of the notes. A real-time note taking system provides a comprehensible index into the recording of the meeting. A user can retrieve and play a part of the recording simply by selecting a corresponding element of the notes.
The simplest example of a real-time note taking system is a text transcription system. The user of a text transcription system simply types the text as the meeting proceeds. The text can be a literal transcription of what is being said or can be arbitrary notes about the meeting. The notes include a string of text, and the time stamped note elements are the individual characters of the text. The notes can also be "strokes", "ink", "digital ink" or "marks" input by a pen based input device. Each stroke of the pen on the display generates a corresponding digital ink mark. The digital ink marks are time-stamped and each mark serves as an index element. After the note taking is complete, the user can select a character or element and the corresponding part of the meeting record will be retrieved and played.
Conventional note taking systems allow users to construct a visual representation of indices into the meeting recording. The visible representation in this case are the notes. Each index element has a time associated with it, which is the time when the element was created. Since each index element is visible, the user selects a character by simply pointing to it. Because the character has an associated time, the user obtains access to the corresponding point in the meeting recording.
Real-time note taking systems work as long as the visual representation is formed by a sequence of element-creating events. However, problems arise when the representation is edited during the course of a meeting. For example, a user may delete and/or change events. However, a deletion removes the element and the associated time stamp. When a new element is substituted for the deleted element, the time stamp of the new element reflects the time that the new element was added, rather than the time of the original element. Therefore, the corresponding time of the underlying event is lost. This occurs because the indices are the visible elements and not the underlying events in the meeting.
Another disadvantage of the conventional real-time note taking system is that a person must be devoted to the note-taking task. It is desirable to create an index into the meeting recording without dedicating a person to the task of note taking. The indexing should be a byproduct of the meeting or activity.
Historical data is captured by multimedia document systems using one or more capture devices, such as an audio recorder or an electronic whiteboard. These recordings typically have timestreams and events. Timestreams are sets of time-stamped continuous data streams which represent some recordable aspect of a meeting. Typical timestreams include, for example, audio and video signals, and/or a history of the inputs on an electronic whiteboard. Events are occurrences within a timestream. For example, an event may be a particularly significant action taken on the electronic whiteboard, such as an input of typed or handwritten notes, page changes, highlighted items, etc., or may be a change in a speaker during the session. Events can be used to create indices which provide direct access to a point in time during a meeting. Timestreams may inherently define events, or alternatively, may be analyzed to identify events. Event information includes a time stamp, an event type and a list of various properties of the instance of the event. Indices into the timestream can be created by a post-session analysis such as by, for example, a speaker identification system analysis on an audio stream.
Documents recorded by multimedia document systems, include, handwriting on an electronic whiteboard, typed or handwritten notes, annotated materials, etc. These documents may have been produced and recorded during a meeting. However, the documents may not have been presented during the meeting but may have been generated by a post-session analysis and recorded as an integral part of the meeting recording.
When reviewing the recording, conventional multimedia document systems provide both the documents and the indices to a user. Typically, a user will read the documents and selectively replay some of the recorded materials. The user may read the documents to understand the content of the meeting and refer to the indices to understand the chronological sequence of events of the meeting.
A problem for user interface design in multimedia document systems is the presentation of the indices to a user. Conventionally, there are two different approaches. The first approach represents the events on a timeline. For example, some systems use colored segments of a timeline to show the speaker during different points in time. However, the user of such a system is required to manage and coordinate at least two different representations. The first representation is the timeline of the events and the second is the display of the documents.
The second approach to user interface representation of indices implicitly represents the indices of the documents in a manner similar to the real-time note taking systems. For example, each stroke drawn on an electronic whiteboard may be time-stamped during the meeting. The time-stamping makes each stroke an index into the recording. However, significant strokes cannot be distinguished from insignificant strokes. Therefore, a user is forced to adopt other methods or systems to keep track of the significant strokes, such as by using, for example, bullets or stars. However, these indices cannot be edited and the history of the recording is not displayed to the user.
Furthermore, multiple times are often associated with a document element, such as the time an element was created, and the time it was last edited. There is no conventional method or system for associating multiple times with index elements.
One multimedia document system uses a clock icon as a temporal icon and index into a recording. Each clock icon indicates a single point in time. Tapping on a clock icon causes the document to start play back at the time indicated on the clock icon. However, these clock icons only indicate a single point in time. In particular, they do not indicate spans of time. Therefore, a user cannot determine how long a specific topic was addressed during a meeting. A user of this multimedia document system only knows at what point in time the discussion of an issue may have been initiated.
Additionally, these clock icons do not indicate the portions of the recording which have been reviewed and those portions which have not been reviewed. Therefore, a user might replay portions of a document that have already been reviewed when the user is intending to review only previously unreviewed portions of the recording. The conventional clock icon indicates the point in time where a discussion of an issue was initiated. However, it does not indicate the progress of the review through the span of time in which the issue was discussed. Therefore, a user is forced to speculate on the progress of a review through a portion of the document.
The conventional clock icons also do not provide any control over playback, other than the ability to initiate the playback at the point in time associated with the clock icon. A user is required to access a separate graphical user interface (hereafter GUI) that resembles a conventional tape deck or VCR control panel.